README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# twitterapiv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This library offers access through Scala to the twitter API v2 endpoints. Additional aids for pagination and throttle management are also provided."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import $ivy.`com.typesafe.akka::akka-http-spray-json:10.2.4`\n",
    "import $ivy.`com.typesafe.akka::akka-slf4j:2.6.8`\n",
    "import $ivy.`com.typesafe.akka::akka-actor-typed:2.6.8`\n",
    "import $ivy.`com.typesafe.akka::akka-stream:2.6.8`\n",
    "import $ivy.`com.typesafe.akka::akka-http:10.2.4`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please, create the project `jar` by typing in the sbt prompt:\n",
    "\n",
    "`sbt:twitterapiv2> package`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import $cp.target.`scala-2.13`.`twitterapiv2_2.13-0.1.jar`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports and dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scala.concurrent.{Future, Await, ExecutionContext, duration}, duration._\n",
    "import _root_.akka.actor.typed.ActorSystem\n",
    "import _root_.akka.actor.typed.scaladsl.Behaviors\n",
    "import scala.util.Success\n",
    "import scala.util.Failure\n",
    "import dev.habla.twitter.{v2_akka, v2}, v2_akka._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Common dependencies for actor-based systems and Akka stream:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "implicit val system = ActorSystem(Behaviors.empty, \"TwitterV2\")\n",
    "implicit val ec = system.executionContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain the bearer token from the environment. Remember to pass this variable to docker, in case you started this notebook from there (`$ docker run -e <var_name> ...`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bearerToken = scala.util.Properties.envOrElse(\"BEARER_TOKEN\", \"undefined\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access programmatically different endpoints to obtain the response of a single request. For instance, we can create a [search/recent](https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent) request and obtain its response as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val request = v2.recents.SingleRequest(\"scala3\", bearerToken, max_results=Some(10))\n",
    "\n",
    "val response: Future[v2.recents.SingleResponse] = request.single"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, we obtain 10 tweets per response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.map{ case v2.recents.Tweets(v2.recents.Tweets.Body(Some(tweets), _, _), _, _) => tweets.map(_.text) }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "show(Await.result(response, 1.second))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can [lookup a tweet](https://developer.twitter.com/en/docs/twitter-api/tweets/lookup/api-reference/get-tweets-id) by its identifier as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2.lookupt.Request(\"787969995917656064\", bearerToken)\n",
    "        .single"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pagination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import akka.stream.scaladsl._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last search for recent tweets, the `next_token` field of the response was not empty: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.map{ case v2.recents.Tweets(v2.recents.Tweets.Body(_, _, meta), r, s) => (meta, r, s) }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that we can issue a new request to get the next page of results (with another 10 tweets): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val next_request = request.copy(next_token = Some(\"b26v89c19zqg8o3fpzbjo0w1eqyrda77g2ag21zxdspkt\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "next_request.single"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This process can go on until we retrieve all the tweets from the response. To do this, we must take into account that there is a maximum number of requests that we can issue to the twitter API in a 15 minutes window. This library can help us with all this pagination and throttle management, by creating a _stream_ of responses instead of a single one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "val response: Source[v2.recents.SingleResponse, akka.NotUsed] = \n",
    "    v2.recents.SingleRequest(\"scala3 -is:retweet\", bearerToken)\n",
    "        .stream"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.take(2)\n",
    "    .toMat(Sink.seq)(Keep.right) // RunnableGraph\n",
    "    .run                         // Future"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this stream, we may take the last `N` tweets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def takeNTweets(request: v2.recents.SingleRequest)(n: Int): Future[Seq[v2.Tweet]] = \n",
    "    request.stream\n",
    "        .mapConcat{ \n",
    "            case v2.recents.Tweets(v2.recents.Tweets.Body(Some(tweets), _, meta), _, _) => \n",
    "                tweets \n",
    "            case _ => List()\n",
    "        }.take(n)\n",
    "        .toMat(Sink.seq)(Keep.right)\n",
    "        .run\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "takeNTweets(v2.recents.SingleRequest(\"scala3\", bearerToken, tweet_fields = Some(\"id,text,created_at\")))(5).map(_.map(_.text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Scala",
   "language": "scala",
   "name": "scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".sc",
   "mimetype": "text/x-scala",
   "name": "scala",
   "nbconvert_exporter": "script",
   "version": "2.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}