Create PostPairsTrading.md #11
Conversation
…ations, extended tips and references It still lacks gifs and images of the assests
There was a problem hiding this comment.
Overall ok, but there were some small spelling mistakes.
The most important thing is that if you decide to use a set nomenclature for something, i.e. "Pairs Trading", keep it consistant, it can't be "pairs trading" one time, then "Pair trading" the next one, etc...
Same goes for "pykx", but in this case, it should always be "PyKX". I have noticed your GIFs has this mixed up as well so please fix them.
If this is going to our blog, all code mentions outside code blocks shoud be done using backticks "`" surrounding them. I'm not sure I have pointed to all of them, so please go through the post line by line and check for this.
Also, I'm not sure how the blog handles markdown quotes (lines beginning with > ). At the moment we can keep those, but we may need to change it later on before merging to the main branch on our blog.
Also, shouldn't it be "kdb+/q" or something similar? I have never seen "Q/kdb+" in the wild. Please make this change.
I have most likely missed some things, so I may need to give it a quick review later on.
|
|
||
| Be sure to stay tuned for more posts and updates on this blog to deepen your knowledge even further. | ||
|
|
||
| Special thanks to [...] for [...] |
There was a problem hiding this comment.
Complete the "special thanks", don't forget Javier.
PostPairsTrading.md
Outdated
|
|
||
| For now, it's crucial to clarify **our spread formulation and understand what it represents**. With this knowledge, we can identify instances where one asset is overpriced while the other is underpriced. | ||
|
|
||
| One might argue that our calculations are heavily influenced by past data, and that we rely too much on historical changes that **may not accurately reflect the present reality**. This is indeed a **valid concern**. To address this issue, we can utilize **the Kalman Filter**, a mathematical method for filtering noise and predicting states in a dynamic system. But we'll delve into the Kalman Filter in our upcoming posts as previously mentioned. |
There was a problem hiding this comment.
This is the second reference to a future Kalman filter post in this section. I think there's some redundancy with it.
There was a problem hiding this comment.
This is true but I am figuring out how to put it. Because next post will explain two different things regarding pairs trading:
- Signal windows
- Kalman filter
PostPairsTrading.md
Outdated
|  | ||
|
|
||
|
|
||
| The graphs illustrate the concept of cointegration between two indexes. The top two graphs show the prices of SP500 (left) and NASDAQ100 (right) over the same time period. We can observe that the price movements of these two indices follow similar patterns, suggesting some level of cointegration. |
There was a problem hiding this comment.
Could we (very briefly) clarify it in the text?
PostPairsTrading.md
Outdated
|
|
||
| The graphs illustrate the concept of cointegration between two indexes. The top two graphs show the prices of SP500 (left) and NASDAQ100 (right) over the same time period. We can observe that the price movements of these two indices follow similar patterns, suggesting some level of cointegration. | ||
|
|
||
| The bottom graph displays the prices of both indices together, providing a clearer comparison. The blue line represents SP500, and NASDAQ100 is represented by the red line. The close alignment of their price movements indicates that they are cointegrated to some extent. This means that, despite short-term deviations, the indices tend to move together as time goes on, maintaining a stable relationship. |
There was a problem hiding this comment.
Not sure if I already mention this in a previous review. What do we consider short-term here? The graphic we are showing is just 15 seconds or so.
There was a problem hiding this comment.
We've added an image that we think clarifies everything here.
| @@ -0,0 +1,247 @@ | |||
| # A Match Made in Trading: Step-by-Step Pairs Trading Guide | |||
There was a problem hiding this comment.
Are you ok with this title?
|
|
||
| Using this approach, we will end up with something like this: | ||
|
|
||
|  |
There was a problem hiding this comment.
This isn't loading properly in Github.
There was a problem hiding this comment.
I still can't see this, so can't review.
PostPairsTrading.md
Outdated
| spread: priceY[.streamPair.i][`bid] - ((priceX[.streamPair.i][`bid] * beta_lr)+alpha_lr); | ||
| ``` | ||
|
|
||
| > 💡 You may notice that we retrieve bid price data from our price stream using an index (`.streamPair.i`). This occurs because we simulate the arrival of these records dynamically, based on a delta time, and thus read from our real-time simulated table, utilizing our updated index `.streamPair.i` with each new record. |
There was a problem hiding this comment.
I think I understand what is going on here, but it's hard to follow.
PostPairsTrading.md
Outdated
|
|
||
| Now that we have selected a pair of cointegrated indices and understand how to calculate their relationships, let's see how we can create a real-time pair trading scenario. | ||
|
|
||
| > ⚠️ An important note is that this post will include a real-time simulation. In other words, if we wanted to develop a 100% real-time product, we would need to make slight adjustments to the code. |
There was a problem hiding this comment.
Does this post include a real-time simulation? I think we aren't discussing that further on, I mean, we aren't showing data here. Maybe you just wanted to refer to the repo.
There was a problem hiding this comment.
I think its more clear now
There was a problem hiding this comment.
Could we move this warning further on? Later, you mention again that this is a simulation ("In our case, we are simulating real time; we do not have a 100% real-time product.") and I feel it's redundant. Could we integrate both references in the same paragraph?
PostPairsTrading.md
Outdated
|
|
||
| After all this, you should be able to understand **our spread formulation and what it represents**. Basically, we can identify instances where one asset is overpriced while the other is underpriced. | ||
|
|
||
| One might argue that our calculations are heavily influenced by past data and that we rely too much on historical changes that **may not accurately reflect the present reality**. This is indeed a **valid concern**. To address this issue, we could implement **a rolling window approach** where the linear regression is continuously updated. |
There was a problem hiding this comment.
I think that we start here with future work, I'd move it to the conclusions.
|
|
||
| Using this approach, we will end up with something like this: | ||
|
|
||
|  |
There was a problem hiding this comment.
I still can't see this, so can't review.
PostPairsTrading.md
Outdated
| resY: priceY[.streamPair.i]; | ||
| s: resY[`bid] - alpha_lr+resX[`bid] * beta_lr; | ||
| enlist `dateTime`spread`mean! | ||
| ("p"$(resX[`dateTime]);"f"$(s);"f"$(0)); |
There was a problem hiding this comment.
So mean is always 0? Do we need parentheses?
I think it's ok to have a simplification but I don't get the objective of this function, I mean, it just returns a dictionary but nobody is collecting the result into a table or sending it to the dashboard. Maybe I'm getting it wrong.
There was a problem hiding this comment.
Yes, when i simplificated i missed the part where we publish. I've just added it :)
PostPairsTrading.md
Outdated
|
|
||
| A simple approach to window signals is to set these windows as twice the historical standard deviation of the spreads. Therefore, if either of these limits is reached, we should sell the overvalued index and buy the undervalued one, and then unwind our position when the spread returns to 0. Let's clarify this with a specific example: | ||
|
|
||
|  |
There was a problem hiding this comment.
This is very nice! 👍
Just one minor thing, what are the values in the X axis?
There was a problem hiding this comment.
The x-axis should represent time, but we struggled to plot in real time because it froze when we tried to edit the x-axis labels.
PostPairsTrading.md
Outdated
|
|
||
| Now that we have selected a pair of cointegrated indices and understand how to calculate their relationships, let's see how we can create a real-time pair trading scenario. | ||
|
|
||
| > ⚠️ An important note is that this post will include a real-time simulation. In other words, if we wanted to develop a 100% real-time product, we would need to make slight adjustments to the code. |
There was a problem hiding this comment.
Could we move this warning further on? Later, you mention again that this is a simulation ("In our case, we are simulating real time; we do not have a 100% real-time product.") and I feel it's redundant. Could we integrate both references in the same paragraph?
PostPairsTrading.md
Outdated
|
|
||
| KDB+/Q stands out as **a powerful tool in finance**, renowned for its ability to handle vast volumes of real-time data amidst the relentless dynamics of the market. In this article, we embark on an insightful exploration of Pairs Trading and its implementation in Q, offering a comprehensive guide to one of the most popular strategies in the trading world. | ||
|
|
||
| Our objective is to **provide an easy-to-understand explanation of some of the intricacies of pair trading**, bridging the gap between theory and practice. |
There was a problem hiding this comment.
This is a derived objective, but not the main one. With the new perspective, the main objective is:
- Implementing Pairs Trading strategy in q
- Identify cointegrated indexes
- Implement model to calculate spreads
- Get it working in real-time
- Integrate with tick architecture
PostPairsTrading.md
Outdated
|
|
||
|  | ||
|
|
||
| As you can see, there are a few new components added to the picture. The details will be discussed as we go on, but I'm sure you can already recognize some familiar faces. |
There was a problem hiding this comment.
I think we should assume that this is their very first contact with kdb (so they shouldn't recognise elements from the diagram). Here I would clarify that this article is aimed at quants in order to motivate them to adopt kdb+/q, due to its incredible performance and concision. But also, we'll supply a brief introduction to Pairs Trading so other developers can benefit from it as well.
|
|
||
| 1. They have a similar trend, meaning the difference between both assets maintains a constant mean, and this difference fluctuates around that same mean. | ||
|
|
||
| 2. This inherent relationship persists in the long run, meaning that our series is not dependent on time. |
There was a problem hiding this comment.
I think that we should add a new section, which supplies a very brief intro to the components of the classical Tick Architecture. I would end up clarifying the next section. First, we'll identify cointegrated pairs. Then, we'll produce the Pairs Trading model to calculate spreads. Finally, we'll exploit such model from a real-time component and will visualise its outcomes.
PostPairsTrading.md
Outdated
|
|
||
| ## A pair in the hand is worth two in the bush | ||
|
|
||
| ### ADF testing |
There was a problem hiding this comment.
Now, this section becomes "Identifying cointegrated indexes" or something like that...
PostPairsTrading.md
Outdated
|
|
||
| **P-values** help us decide whether to reject the null hypothesis. If the p-value is low, it indicates that we can reject the hypothesis that the time series is non-stationary, suggesting that our assets are cointegrated. The lower the p-value, the greater the confidence in rejecting the null hypothesis. It is very common to use a threshold of 0.05 on the p-value to reject hypotheses. | ||
|
|
||
| For the sake of simplicity, we will be using [PyKX](https://code.kx.com/pykx/2.4/index.html). This is necessary as we require importing our ADF test function and plotting a heatmap of our results. Developing these functionalities directly in Q would be time-consuming and prone to errors. Although an implementation of the ADF test in KDB+/Q would be more efficient and faster, the effort required would outweigh the benefits. Therefore, we rely on PyKX to streamline the process by leveraging relevant libraries from the Python ecosystem. |
There was a problem hiding this comment.
... outweigh the benefits for this particular case where performance isn't critical.
There was a problem hiding this comment.
Why don't we import pykx right here as we did before?
PostPairsTrading.md
Outdated
|
|
||
| > 💡 This strategy possesses financial characteristics: our **profitability remains unaffected by the broader market trends**, as our focus lies solely on the disparity between the two assets. It's about relative movements rather than absolute ones; we're indifferent to whether prices are rising or falling. This quality defines it as a **neutral market strategy**. | ||
|
|
||
| ## Spreading spreads |
There was a problem hiding this comment.
Ok, now let's go for the Pair Trading model that calculates spreads. Could we find a nicer name for this section?
| alpha_lr:alphaF[t[`SP500]`close;t[`NASDAQ100]`close] | ||
| ``` | ||
|
|
||
| This precisely meets one of our objectives: getting **a comprehensive method for representing relative changes between both assets**. As we can deduce, our mean is now 0 because our assets are normalized, cointegrated and on the same scale. Therefore, ideally, the differential between their prices should be 0. Consequently, when our spread is below 0, we infer that asset X is overpriced, whereas if it's above 0, then asset Y is overpriced. |
There was a problem hiding this comment.
Again, we should rely on the diagram here. We can emphasise the HDB, the line that reads from the HDB and the RPT (but not emphasising dashboards).
| ```q | ||
| .z.ts: {.u.pub[`spreads;update priceY - alpha_lr+priceX * beta_lr from prices]} | ||
| \t 100 | ||
| ``` |
There was a problem hiding this comment.
I miss the pub method from the RPT component itself, where you accumulate on the prices table. Which component is responsible for cleaning the published spreads?
PostPairsTrading.md
Outdated
|
|
||
| And there we have it! **A perfectly plotted spread series in real-time**, ready to be utilized for further analysis and exploitation. | ||
|
|
||
| ## What's left to start making a profit? |
There was a problem hiding this comment.
I'm not sure if we should create another section for this, maybe we could place this at the end of the previous one.
|
|
||
| > 💡 Signal windows play a pivotal role in implementing Pairs Trading strategies. They serve as indicators for determining when to execute buy and sell actions, acting as arbitrary thresholds that guide our algorithm's decision-making process. These windows are derived from the variance of our data, representing a static variance assumption due to our consideration of a time-independent cointegrated series. | ||
|
|
||
| ## Conclusion and Future Work |
There was a problem hiding this comment.
The content is fine, but requires some reordering. We should explicitly address the objectives that we stated in the introduction. Future work is fine.
|
|
||
| We will be covering the following aspects using the KDB tick architecture as a base: | ||
|
|
||
|  |
There was a problem hiding this comment.
Very important: we need to add a reference to the original picture from DefConQ
|
|
||
| ```q | ||
| pyhm:.pykx.import[`seaborn]`:heatmap | ||
| pyhm[pvalues;`xticklabels pykw syms;`yticklabels pykw syms;`cmap pykw `RdYlGn_r] |
There was a problem hiding this comment.
I'd add a very short sentence describing what is going on here. I'm unable to decipher the RdYlGn_r.
| In our case we are going to use closing prices to our ADF test, so we have to index (`@`) by column **close** from each pair in our table. Additionally, we take (`#`) the last **tr** days of data for both indexes, and finally apply our **fcoint** function to each (`.'`) pair of data lists. | ||
|
|
||
| ```q | ||
| matrix:fcoint .' 0f^@\:[;`close](@/:[t]')syms cross syms |
There was a problem hiding this comment.
Would it make sense to just select sym and close in rs? (Take a look at my proposed text #13 to understand why)
| We simply need to declare an `sp` function that calculates the spread given the prices, but this is something we already know how to do. | ||
|
|
||
| ```q | ||
| sp:{y - alpha_lr + beta_lr * x}; |
There was a problem hiding this comment.
This should be defined in the previous section, since the model is an input for this step. I prefer hiding alphas and betas in this section. What do you think?
|
|
||
| Be sure to stay tuned for more posts and updates on this blog to deepen your knowledge even further. | ||
|
|
||
| Special thanks to [...] for [...] |
There was a problem hiding this comment.
Complete the "special thanks", don't forget Javier.
|
|
||
| > 💡 This strategy possesses financial characteristics: our **profitability remains unaffected by the broader market trends**, as our focus lies solely on the disparity between the two assets. It's about relative movements rather than absolute ones; we're indifferent to whether prices are rising or falling. This quality defines it as a **neutral market strategy**. | ||
|
|
||
| ## Determining how to calculate the spreads |
There was a problem hiding this comment.
I want to discuss two aspects from this section with you. Let me know when you find the time! Just as a reminder for myself:
- HDB -> RPT link
- price_y and price_x source (rs?)
Description:
This pull request aims to create a Markdown file for a post on pairs trading, aggregating all the content previously developed. The post will serve as a comprehensive guide to pairs trading, covering various aspects such as the concept, rationale, implementation, and practical examples.
Changes Proposed:
Create a new Markdown file named "PostPairsTrading.md".
Structure the Markdown file with appropriate headings and sections, including:
-Rationale and Concept
-Methodology and Implementation
-Practical Examples
-Conclusion
Incorporate content from previous materials, including explanations, code snippets, and visualizations, ensuring clarity and coherence.
Review and proofread the content to ensure accuracy, consistency, and conciseness.
Include references or citations for any external sources or materials used in the post.
Purpose:
The purpose of this pull request is to compile and organize the existing materials on pairs trading into a cohesive and informative Markdown file. By creating a structured and well-documented post, we aim to provide readers with a comprehensive resource for understanding and implementing pairs trading strategies effectively.