On the journey that led me from a prototype to a local, modular and reliable desktop architecture, I separated the presentation layer, application/domain layer, and persistence.
But a well-designed architecture is not enough.
If the data flowing through it is incomplete, duplicated, or misinterpreted, everything built on top of it simply becomes a more sophisticated way of producing the wrong results.
And this is precisely where, in the project I am developing, Data Engineering stopped being a supporting phase and became a central part of product reliability.
Raw Data Is Not Yet Information
The starting point is a set of text files produced by the platform after each hand.
At first glance, they seem simple.
They contain:
- a hand identifier;
- the tournament;
- the level;
- blinds and antes;
- players;
- stacks;
- positions;
- cards;
- actions;
- the board;
- the showdown;
- winners;
- the pot.
But the problem is not reading lines of text.
The problem is transforming a textual representation designed primarily to be read by a human into a structured, consistent, and queryable model.
In other words:
text history ≠ structured data
A genuine interpretation pipeline is needed between those two ends.

A Parser Is Not Universal: It Interprets a Specific Format
At first, it is easy to think of a parser as a collection of regular expressions.
Find a line.
Extract a few values.
Move on to the next one.
That works while the data is simple.
Then the real cases arrive.
A hand may end before the flop. There may or may not be a showdown. A player may be all-in. There may be a side pot. A bet may be returned because no one covers it. The winner may or may not show their cards.
The structures vary slightly depending on the tournament, its stage, and the events in the hand.
At that point, the parser is no longer simply reading text.
It is reconstructing domain state.
There is, however, an even more important aspect.
The parser developed today adheres to the type of hand history chosen as its reference format.
This means it understands:
- the structure of the sections;
- how actions are represented;
- the syntax of amounts;
- how streets are described;
- the summary format;
- how winners, showdowns, side pots, and other events are expressed.
This knowledge is intentional.
A reliable parser does not emerge from trying to interpret any generic “poker text”.
It emerges from understanding a precise format and transforming it into a coherent domain model.
This is why I consider it important to distinguish between the poker domain and the source format of the hand history.
The application domain can remain stable.
The parser, by contrast, is necessarily tied to the structure of the input data.
If the Domain Expands, the Parser Must Evolve Too
This distinction has an important architectural consequence.
If the product were to support hand histories from other poker rooms tomorrow, I would not expect the current parser to interpret them automatically.
Headers, section names, action representations, stack syntax, blind descriptions, tournament formats, showdown handling, summary structures, edge cases, and metadata could all be different.
Expanding the product to new sources would therefore not simply mean:
“let’s add another file to import”.
It would require the ingestion layer to evolve.
The model I consider most appropriate is:
source-specific parser → normalised domain model
Each source format can have its own interpretation logic, but they all converge on the same internal representation.
Conceptually:
Room A Hand History → Parser A
Room B Hand History → Parser B
Room C Hand History → Parser C
all feeding into:
Normalised Poker Domain Model
This distinction is fundamental to keeping the product evolvable.
The parser must be specialised. The domain should remain as independent of the source as possible.
The Parser Is Therefore a Domain Interpreter
A line indicating a bet, for example, is not interesting only because of the amount.
We need to know:
- who acted;
- on which street;
- what type of action they took;
- what the amount represents;
- where it falls in the sequence of events;
- which players were still active at the time;
- what the state of the pot was.
This is where parsing and domain modelling inevitably begin to meet.
But the objective is to prevent the domain from accidentally inheriting the syntactic peculiarities of the source.
The parser translates.
The domain represents.
This separation becomes even more important as the number of supported sources grows.
Separating Parsing from Persistence
One important decision was to prevent the parser from writing directly to the database.
Instead, the conceptual flow is:
file → source parser → domain objects → persistence
The parser must know how to interpret the hand history.
It must not know how it is stored.
Persistence must know how to store a structured hand.
It must not know how that hand was originally written in the file.
This separation offers an enormous advantage during development.
When a new hand history variant emerges, I can work on the parser without changing the repositories or database.
And when the persistent structure changes, I can work on persistence without contaminating the parsing rules.
In future, the same separation would make it possible to add a parser for a new source without rewriting everything that follows.
It is the same architectural principle that also runs through From Prototype to Product:
each component should know as little as possible about the others.
Normalise Before Analysing
The second problem arrives immediately after parsing.
Even when two elements conceptually represent the same thing, they may not arrive in the same format.
Dates. Amounts. Positions. Streets. Action types. Tournament metadata. Board. Results.
Everything entering the system must be transformed into a coherent representation.
Because analytics needs to be able to assume that:
the same information always has the same meaning and the same structure.
This normalisation is important even with a single source.
It becomes even more important if multiple sources are supported in future.
Two poker rooms may describe the same action differently.
The system must nevertheless represent it in the same way.
This means that the real contract is not the source text. It is the internal normalised model.
If a position is interpreted differently in two hands, a positional statistic may be formally correct but substantively wrong.
If a raise and an all-in raise are classified inconsistently, the problem will resurface later in aggression statistics.
If a street is assigned incorrectly, the error can propagate into CBet, Fold to CBet, aggression frequency, and dozens of other metrics.
The important point is that the most dangerous error is not always the one that causes an exception.
Often, it is the one that produces data that is perfectly valid from a technical perspective but semantically wrong.
Validation Does Not Only Mean “The File Is Readable”
For this reason, I progressively separated two concepts:
parsing success
and
data correctness
A hand can be read without producing errors and still contain an incorrect reconstruction.
Validation must therefore check domain invariants.
For example:
- a hand must have an identifier;
- it must have a coherent set of players;
- actions must belong to valid streets;
- the board must respect the flop/turn/river progression;
- winners must be consistent with the hand structure;
- amounts must be representable correctly;
- the information required for statistics must not be lost during transformation.
This significantly changes the concept of testing.
I am no longer checking only:
“Does the parser complete without errors?”
I am checking:
“Does the resulting representation genuinely preserve the meaning of the hand?”

The Invisible Problem of Duplicates
There is also a less glamorous but fundamental problem: the same hand may arrive more than once.
The user may accidentally import the same file, a folder that has already been processed, copies of the same history, or partially overlapping files.
If every import produced new rows, all statistics would progressively become unusable.
A frequency observed twice does not become more reliable.
It simply becomes duplicated.
For this reason, deduplication is not a detail of the import interface.
It is a property of the data model.
In the database, hand identity is protected through a unique key based on the source and the external hand identifier.
Conceptually:
UNIQUE(source, external_hand_id)
This formulation is also important from a multi-room perspective: the same numerical identifier could theoretically exist in different sources.
The principle is:
idempotency must be guaranteed by the system, not remembered by the user.
If I import the same dataset twice, the final result must remain consistent.
From Files to Persistent Entities
A hand is not stored as one enormous blob of text.
It is broken down into queryable elements.
In the persistent model, distinct responsibilities exist for:
- the hand;
- the players in the hand;
- actions;
- winners;
- the imported file.
This enables something fundamental for everything that follows:
reconstructing metrics from events.
A statistic such as Fold to 3Bet is not stored inside the hand.
It is derived from position, the preflop sequence, the opportunity, and the player’s action.
The same applies to CBet, steal, blind defence, aggression, and many other metrics.
Persistence therefore does not need to anticipate every future statistic.
It must preserve enough atomic information to calculate them later.
This was one of the most important decisions in the data model.
Preserve Events, Not Conclusions
It is a principle I find particularly useful:
where possible, persist facts and derive interpretations.
An action is a fact.
A board is a fact.
A winner is a fact.
A position is a fact reconstructed by the domain.
“This player overfolds” is an interpretation.
If I store only the interpretation, it becomes difficult tomorrow to change the formula, correct a definition, introduce a new statistic, show the supporting hands, compare a player with the population, or assess confidence and sample size.
If I preserve the events, however, I can recalculate the conclusions.
This distinction is fundamental for a product that aims to evolve from statistics to intelligence.

Testing with Real Data
A parser can have a perfect suite of unit tests and fail on the first real folder.
For this reason, once the parser had reached a certain level of maturity, I added a separate verification based on real tournament hand histories.
At that point, the corpus comprised six real files from the supported source.
The result of the latest parser verification was:
404 hands analysed
404 parsed successfully
0 failures
On a file containing 117 hands, the parser completed processing in approximately 72 ms, with a throughput of around 1,600 hands per second in the test environment.
Performance, however, was not the most important part of that result.
What mattered was that the parser was being verified against material that had not been constructed specifically to make the test pass.
It is a different form of quality.
Unit tests protect the cases I know.
The real-world corpus helps uncover those I had not imagined.
404/404 Does Not Mean “Universal Parser”
This is another important point.
A parser that passes 100% of the current corpus is not necessarily complete.
It means:
it correctly interprets every variant observed up to that point for the supported format.
It does not mean:
“any hand history produced by any poker room can be interpreted automatically.”
Nor does it mean that the same source cannot introduce new variants in future.
There may still be new summary structures, tournament descriptors that have not yet been observed, unusual formats, rare events, histories from different configurations, and syntactic differences between poker rooms.
For this reason, I consider the parser a component that must be monitored and evolved alongside the input domain.
When the supported sources change, the ingestion problem changes too.
And this is not an architectural flaw.
It is a natural consequence of the fact that the source data is part of the integration contract.
Reliability Is an End-to-End Property
Ultimately, this work reminded me that data quality does not belong to a single component.
It is a chain.

If I misinterpret the hand history:
→ I store the wrong data.
If I store the wrong data:
→ I calculate the wrong statistics.
If I calculate the wrong statistics:
→ I generate the wrong patterns.
If I generate the wrong patterns:
→ every layer of intelligence built on top will be wrong.
The more sophisticated the system becomes, the more important this problem becomes.
Because a sophisticated algorithm applied to the wrong data does not fix the problem.
It simply makes it more convincing.
Trust Comes Before Intelligence
This is perhaps the most important conclusion.
When we build systems that transform data into decision support, we are naturally drawn to the final layer:
AI.
Pattern recognition.
Prediction.
Intelligence.
But the value of the final layer depends entirely on how much we can trust the first one.
For this reason, before building an engine capable of saying:
“This player exhibits this behaviour”
I want to be reasonably certain that the system has correctly interpreted who the player was, where they were seated, what they did, when they did it, in relation to which opportunity, and how many times that situation actually occurred.
And I want this to remain true when, in future, the product begins receiving data from different sources.
Because increasing the number of supported sources does not simply mean increasing the number of readable files.
It means preserving the same level of reliability across different parsers that converge on a single domain model.
But reliable data still does not solve everything.
A statistic can be mathematically correct and, at the same time, completely misleading if we do not know how many opportunities generated it, how large the sample is, and how much confidence we can place in it.
That is the next step in this journey: when the number alone is not enough.
Originally published on LinkedIn on 23 August 2026. This edition is part of Lead From Tech Insights.
