Read one level up: https://search.qdrant.tech/md/documentation/tutorials-operations

> Explore Qdrant's agent skills catalog at https://skills.qdrant.tech/
> Search the documentation at https://skills.qdrant.tech/search?query=your+query+here
> Use this file to discover all available pages: https://qdrant.tech/llms.txt


# Time-Based Sharding in Qdrant

When working with massive, fast-moving datasets, like social media or image/video streams, efficient storage and retrieval are critical. Often, only the most recent data is relevant, while older data can be archived or deleted. For instance, in sentiment analysis of social media posts, you might only need the last 7 days of data to capture current trends, with most queries focusing on the last 24 hours.

Storing everything in Qdrant collection with default sharding can lead to expensive re-indexing across the entire dataset when deleting old points, impacting performance. A better solution is **time-based sharding**, where points are routed to a specific [shard (or shards)](https://qdrant.tech/documentation/scaling/distributed_deployment/index.md#sharding) based on timestamp. For use cases with a natural time-to-live (TTL) segmentation, sharding by a timestamp-based key enables efficient querying of recent data and allows users to seamlessly drop the old.

For example, with daily shards, today's data is stored in today's shard, yesterday's data in yesterday's shard, and so on. Queries can target specific shards (today's shard, for example) or multiple shards to cover a date range.

![Time-based sharding across daily shards](/md/documentation/tutorials/time-based-sharding/time-based-sharding.png)

*Time-based sharding routes data to shards based on timestamp. All writes go to the newest shard, while queries can target one or more shards. Older shards can be pruned in the background without affecting performance.*


Depending on your data volume and retention needs, you could shard by hour, week, month, or any other time interval that suits your use case.

This tutorial guides you through implementing time-based sharding and covers:

* Creating a Qdrant collection with user-defined sharding
* Batch-ingesting historical data into the correct shards based on timestamps
* Assigning new data to the most recent shard
* Querying one or more shards
* Pruning older shards

The interval and the retention window are the two choices this tutorial makes for you. The Plan time-based shards prompt has an agent derive both from your own data and query pattern, and say when the approach does not fit.


> Prompt: Plan time-based shards
> Related agent skill: https://skills.qdrant.tech/qdrant-scaling/scaling-data-volume/sliding-time-window/SKILL.md


## Install and Initialize the Qdrant Client

First, install the Qdrant client:

```bash
pip install qdrant-client
```

Next, initialize the client: 

```python
from qdrant_client import QdrantClient, models

client = QdrantClient(
    url=QDRANT_URL,
    api_key=QDRANT_API_KEY,
    cloud_inference=True
)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/initialize-client/index.md).


This tutorial assumes you are using [Qdrant Cloud Inference](https://qdrant.tech/documentation/inference/cloud-inference/index.md) to generate vector embeddings. If you manage your own embedding infrastructure, you can apply the same principles, but you will need to adapt the code examples to use your embedding service.

## Create Collection

Create a collection with [user-defined sharding](https://qdrant.tech/documentation/scaling/distributed_deployment/index.md#user-defined-sharding) by setting the sharding method to custom.

```python
from qdrant_client import models

collection_name = "my_collection"

if client.collection_exists(collection_name=collection_name):
    client.delete_collection(collection_name=collection_name)

client.create_collection(
    collection_name=collection_name,
    vectors_config={
        "dense_vector": models.VectorParams(
            size=384, distance=models.Distance.COSINE
        )
    },
    sharding_method=models.ShardingMethod.CUSTOM
)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/create-collection/index.md).


Custom shards can be accessed by their shard key. In this tutorial, the shard keys are the dates in `YYYY-MM-DD` format, extracted from the timestamp of each data point.

This collection will have a single shard for each shard key (a separate shard for each day of data). For very large datasets, you can improve write throughput by configuring a `shard_number` for the collection. `shard_number` defaults to 1. Set it to a higher value to create multiple shards per shard key to distribute the write load across multiple peers in the cluster. However, avoid creating too many shards, as each shard consumes resources and adds overhead, which can lead to performance degradation. Test what the optimal number of shards is for your dataset and cluster configuration.

Two things to note about collections that use user-defined sharding versus regular collections using auto sharding:
- For regular collections using auto sharding, `shard_number` determines the total number of shards for the collection. However, with user-defined sharding, `shard_number` determines the number of shards **per shard key**: the total number of shards for a collection equals the number of shard keys (days) multiplied by the `shard_number`.
- Collection-level configuration changes that you can apply to a regular collection (for example, HNSW parameters) can also be applied to a collection with user-defined sharding. These changes are applied retroactively to existing shards and to new shards created in the future.

## Ingest Historical Data

Time series data often arrives in streams, with new data points continuously being added. Each data point may include a timestamp indicating when it was created. You can use this timestamp to determine which shard a data point belongs to. If your data does not have timestamps, you can use the current time.

This tutorial uses a [sample dataset of social media posts](https://github.com/qdrant/examples/blob/master/time-based-sharding/social-media-posts.csv) with timestamps. Let's assume today is April 7th, 2026. You'll start by ingesting some historical data from this week (April 1-7). Here are a few sample rows from the dataset:

| datetime            | text
|---|---|
| 2026-04-06T09:04:28 | April sunshine through the office window makes everything better.
| 2026-04-06T09:04:32 | Morning stretch, good coffee, clear intentions. Monday: sorted.
| 2026-04-06T09:05:52 | Grateful for a productive first day of the week.

Upload the dataset and store each day of data in its own shard:

```python
from qdrant_client.http.models import PointStruct, Document
import uuid

csv_url = 'https://raw.githubusercontent.com/qdrant/examples/refs/heads/master/time-based-sharding/social-media-posts.csv'

# Retrieve a list of existing shard keys in the collection
existing_shard_keys = [
    key.key for key in client.list_shard_keys(collection_name=collection_name).shard_keys or []
]

dense_model = "sentence-transformers/all-MiniLM-L6-v2"
batch_size = 100
current_date = None
buffer: list[PointStruct] = []

for row in parse_csv(csv_url):
    shard_date = row['datetime'][:10]  # Extract YYYY-MM-DD

    if shard_date != current_date:
        # Flush buffer for the previous date before switching
        if buffer:
            client.upload_points(
                collection_name=collection_name,
                points=buffer,
                shard_key_selector=current_date,
            )
            buffer = []

        # Create shard for the new date if it doesn't exist yet
        if shard_date not in existing_shard_keys:
            client.create_shard_key(collection_name, shard_date)
            existing_shard_keys.append(shard_date)

        current_date = shard_date

    # Add point to buffer
    buffer.append(PointStruct(
        id=uuid.uuid4().hex,
        payload={"text": row['text'], "datetime": row['datetime']},
        vector={"dense_vector": Document(text=row["text"], model=dense_model)}
    ))

    # Flush batch if buffer size exceeds batch size
    if len(buffer) >= batch_size:
        client.upload_points(
            collection_name=collection_name,
            points=buffer,
            shard_key_selector=current_date,
        )
        buffer = []

# Flush remaining partial batch
if buffer:
    client.upload_points(
        collection_name=collection_name,
        points=buffer,
        shard_key_selector=current_date,
    )
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/upload-vectors/index.md).


Let's break down the code:

- First, a list of existing shard keys in the collection is retrieved. There should be none because you just created the collection, but in production, this ensures you take into account any existing data.
- Next, a CSV file is streamed from a URL using a helper function to parse the CSV.
  <details>
  <summary>Details</summary> 
  ```python
  import csv
  import urllib.request
  
  def parse_csv(url):
      with urllib.request.urlopen(url) as response:
          reader = csv.DictReader(line.decode('utf-8') for line in response)
          yield from reader
  ```
  
  > This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/parse-csv/index.md).

  </details>
- The CSV file is streamed row by row, buffering points in batches of 100 for efficient uploading. The optimal batch size depends on your data and cluster, so you may want to experiment with different sizes for best performance.
- The date (`YYYY-MM-DD`) is extracted from each row's datetime field. This date is used as the shard key to route the data to the correct shard.
- A new shard is created for each new date encountered if it doesn't already exist.
- The buffer is flushed to the previous date's shard whenever the date changes mid-stream, ensuring posts don't get written to the wrong shard.
- Data is written, where each point gets:
  - A random UUID as the point ID
  - The post `text` and `datetime` as the payload
  - A dense vector embedding generated from the post text using `sentence-transformers/all-MiniLM-L6-v2`
- Each full  batch of 100 points is uploaded to Qdrant, targeting the correct date-based shard via the shard key selector parameter.
- Any remaining points are uploaded in a partial final batch after the loop ends.

## Query the Data

### Query Today's Data

Now you can run a semantic query on the posts. Setting the shard key selector to `2026-04-07`  (assuming today is April 7th, 2026) limits the query to the shard that stores today's data:

```python
query_text = "coffee"

resp = client.query_points(
    collection_name=collection_name,
    query=Document(text=query_text, model=dense_model),
    using="dense_vector",
    limit=5,
    shard_key_selector="2026-04-07"
)
print(resp)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/search-single-shard/index.md).


### Query Multiple Days of Data

To query multiple shards, set the shard key selector to a list of shard keys. For example, to query the last 2 days of data (April 6-7):

```python
resp = client.query_points(
    collection_name=collection_name,
    query=Document(text=query_text, model=dense_model),
    using="dense_vector",
    limit=5,
    shard_key_selector=["2026-04-06","2026-04-07"]
)
print(resp)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/search-multiple-shards/index.md).


### Query the Full Dataset

To query the entire dataset (all shards), omit the shard key selector parameter:

```python
resp = client.query_points(
    collection_name=collection_name,
    query=Document(text=query_text, model=dense_model),
    using="dense_vector",
    limit=5,
)
print(resp)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/search-all-shards/index.md).


## Pruning Shards

Every night at midnight, create a new shard for the new data that will be ingested that day. If you only query the last 7 days of data, you can also delete the oldest shard. You can automate this with a cron job. The example sets `today` to `2026-04-08` to match the sample data. In a scheduled job, use `date.today().isoformat()` instead.

```python
from datetime import date, timedelta

today = "2026-04-08"
oldest_shard_key = (date.fromisoformat(today) - timedelta(days=7)).isoformat()

client.create_shard_key(collection_name, today)
client.delete_shard_key(collection_name, oldest_shard_key)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/pruning-shards/index.md).


## Ingest New Data

When ingesting new data, set the `shard_key_selector` to today's date, using the `today` value defined in the pruning step, so the data goes to the correct shard:

```python
client.upsert(
    collection_name=collection_name,
    points=[PointStruct(
        id=uuid.uuid4().hex,
        payload={"text": "The best way to start a Wednesday is with a cup of coffee", "datetime": "2026-04-08T07:57:47"},
        vector={
            "dense_vector": Document(text="The best way to start a Wednesday is with a cup of coffee", model=dense_model)
        })],
    shard_key_selector=today
)
```

> This snippet is also available in Typescript, Rust, Java, Csharp, and Go. See the [full snippet](https://qdrant.tech/documentation/snippets/time-based-sharding/ingest-new-data/index.md).


## Conclusion

Time-based sharding is a powerful technique for managing large, time-series datasets in Qdrant. By routing data to different shards based on timestamps, you can efficiently store and query recent data while easily pruning older data without impacting performance. This approach is ideal for use cases like social media analysis, where data relevance decreases over time.