A high-throughput, cloud-native market-data ingestion and distribution engine built with Python 3.11, FastAPI, and TimescaleDB. Handles real-time tick data, OHLCV aggregation, and serves pricing microservices via async REST and WebSocket endpoints.
Architecture
- Ingestion layer: async producers consuming exchange feeds and vendor APIs
- Storage: TimescaleDB for time-series, Redis for hot-path caching
- API: FastAPI with SQLAlchemy async, Pydantic v2 validation
- Deployment: Kubernetes, GitHub Actions CI/CD
Key Features
- Sub-millisecond write latency at scale
- Configurable retention and compression policies
- Instrument normalisation across multiple data vendors
- Full audit trail for regulatory reporting