asavinov

Joined 10 years ago

Karma

283

Submissions

129

Member since

Jul 10

https://github.com/asavinov - My projects

https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

View on Hacker News

Recent submissions

Comment · 2 years ago

You could check out a tool for trade signal generation based on machine learning and feature engineering:

https://github.com/asavinov/intelligent-trading-bot

It trains ML models based on historic data and custom features and then uses them to generate a kind of intelligent indicator between -1 and +1. This intelligent indicator is then used to make trade decisions. Frequency is a parameter and can vary from 1 minute for crypto trading to 1 day for normal exchanges.

Comment · 2 years ago

The sequence

    FROM Table AS t WHERE t.Condition SELECT t.col1, t.col2, ...

might be more natural than the traditional

    SELECT t.col1, t.col2, ... FROM Table AS t WHERE t.Condition

If we compare it with how loop are described in programming languages:

    Loop -> SELECT-FROM-WHERE
    Table -> Collection
    AS t -> Loop instance variable
    WHERE -> condition on instances

In Java and many other PLs, we write loops as follows:

    foreach x in Collection
        if x.field == value:
            continue
        // Do something with x, for example, return in a result set

So we first define the collection (table) we want to process elements from. Then we think about the condition they have satisfy by using the instance variable. And finally in the loop body we do whatever we want, for example, return elements which satisfy the condition.

In Python, loops also specify the collection first:

    for x in Collection:

Python list comprehension however uses the traditional order:

    [(x.col1, x.col2) for x in Collection if x.field2 == value]

Here we first specify what we want to return, then collection with condition.

Comment · 3 years ago

From my experience with crypto currencies and the intelligent trading bot [0] I would say that transformers will not provide significant benefits when applied to the traditional statistical (numeric) forecasting problems. Such models assume that old events do not affect too much current events.

Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago.

[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

Comment · 3 years ago

I agree that the conventional (numeric) forecasting can hardly benefit from the newest approaches like transformers and LLMs. I made such a conclusion while working on the intelligent trading bot [0] by experimenting with many ML algorithms. Yet, there exist some cases where transformers might provide significant advantages. They could be useful where the (numeric) forecasting is augmented with discrete event analysis and where sequences of events are important. Another use case is where certain patterns are important like those detected in technical analysis. Yet, for these cases much more data is needed.

[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

Comment · 3 years ago

Does it work synchronously or asynchronously? For example, if you integrate WebSockets then you could act as soon as you get new data. If it works synchronously then you regularly request data (say, every second) and then act.

If you process many positions and then choose top 5 candidates then do you choose one of them for buying or you can allocate resources between them (depending on some score)?

Comment · 3 years ago

When developing an automatic trading system the following aspects are important:

- Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format

- Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work

- Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds.

- Real trading and order management as well as coordination of all activities.

The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels.

I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important

Comment · 3 years ago

> What gives you advantage is trading algo, which is always hard to find.

At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:

https://github.com/asavinov/intelligent-trading-bot

Currently it works for cryptocurrencies but can be applied to other markets:

https://t.me/intelligent_trading_signals

> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.

It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.

Comment · 3 years ago

For any such tool, two questions are of primary importance:

- How connections between multiple tables are represented and managed

- How derived data is described (queries, workflows etc.)

Typically such tools are aimed at simplifying data connections but normally they end up with some kind of join-like approach which requires high expertise and is error prone. So users have to deal with something they wanted to get rid of when they buy the tool. Plato is not an exception: "No SQL needed". Yet, I could not find any information on how exactly it manages connections between tables and how the unified "virtual table" is defined.

The second question is about how we can derive new data from existing data. Ideally, users would like to have something very similar to Excel because spreadsheets are indeed extremely intuitive: we define new cells as functions of other cells (which in turn might be functions of other cells). In Plato I found "virtual columns" which should be rather useful. This is somewhat similar to the column-oriented approach implemented in Prosto [0]. Yet, what is really non-trivial is how to define (derived) columns by combining data from multiple tables.

In general, the tool looks very promising and I hope that additional features and additional information will make it really popular.

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

Comment · 4 years ago

> I have always kept in mind is that feature engineering is almost always the key difference between success and failure

I also developed an ML-powered service heavily relying on feature engineering

https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot

Its difference from Didact is that this intelligent trading bot is focused on trade signal generation with higher frequency of evaluation. It is more suitable for cryptocurrencies but also works for traditional stocks with daily frequencies so it could be adapted for stock picking. What I find interesting in your work is the general design of such kind of ML systems relying on feature engineering.

Comment · 4 years ago

> Joins are what makes relational modeling interesting!

It is the central part of RM which is difficult to model using other methods and which requires high expertise in non-trivial use cases. One alternative to how multiple tables can be analyzed without joins is proposed in the concept-oriented model [1] which relies on two equal modeling constructs: sets (like RM) and functions. In particular, it is implemented in the Prosto data processing toolkit [2] and its Column-SQL language [3]. The idea is that links between tables are used instead of joins. A link is formally a function from one set to another set.

[1] Joins vs. Links or Relational Join Considered Harmful https://www.researchgate.net/publication/301764816_Joins_vs_...

[2] https://github.com/asavinov/prosto data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

[3] Column-SQL https://prosto.readthedocs.io/en/latest/text/column-sql.html

Comment · 4 years ago

One idea is to use columns instead of cells. Each column has a definition in terms of other columns which might also be defined in terms of other columns. If you change value(s) in some source column then these changes will propagate through the graph of these column definitions. Some fragments of this general idea were implemented in different systems, for example, Power BI or Airtable. The main difficulty in any formalization is how to deal with columns in multiple tables.

This approach was formalized in the concept-oriented model of data which relies on two basic elements: mathematical functions and mathematical sets. In contrast, most traditional data models rely on only sets. Functions are implemented as columns. This model gets rid of joins and groupby by making data processing simpler and more intuitive.

This approach was implemented in the Prosto data processing toolkit: https://github.com/asavinov/prosto

Comment · 4 years ago

Looks interesting although I miss a short introduction or how-to guide. I found that one can "Create and train machine learning models to predict market values." In this context, a related project is Intelligent Trading Bot: https://github.com/asavinov/intelligent-trading-bot which is intended for generating trading signals based on ML and feature engineering

Comment · 4 years ago

Having Python expressions within a declarative language is a really good idea because we can combine low level logic of computations of values with high level logic of set processing.

A similar approach is implemented in the Prosto data processing toolkit:

https://github.com/asavinov/prosto

Although Prosto is viewed as an alternative to Map-Reduce by relying on functions, it also supports Python User-Defined Functions in its Column-SQL:

  prosto.column_sql(
    "CALCULATE  Sales(quantity, price) -> amount",
    lambda x: x["quantity"] * x["price"]
  )

In this Column-SQL statement we define a new calculated column the values of which are computed in Python (as a sum of two columns). An advantage is that we can process data in multiple tables without joins or groupbys which is much easier than in the existing set-oriented approaches. Another advantage is that we can combine many statements by defining a workflow in an Excel-like manner.

Comment · 5 years ago