snapshotter

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2025 License: Apache-2.0 Imports: 17 Imported by: 0

README

Snapshotter

License Go Version

Snapshotter is a high-performance file snapshotting and restoration library for Go, designed to work with Dragonfly for P2P-accelerated file distribution. It provides efficient file deduplication using content-addressable storage and supports OCI-compatible registries for metadata management.

Features

  • Snapshot & Restore: Create snapshots of files and restore them efficiently
  • Content-Addressable Storage: Automatic deduplication using content hashing (XXH3)
  • OCI Registry Integration: Store and retrieve snapshot metadata from OCI-compatible registries
  • Dragonfly Integration: P2P-accelerated file distribution for fast downloads
  • Sparse File Support: Efficient handling of sparse files
  • Hardlink-Based Deduplication: Minimize storage usage for read-only files
  • Automatic Garbage Collection: Background cleanup of unused content
  • Concurrent Operations: Safe for concurrent snapshot and restore operations

Installation

go get d7y.io/snapshotter

Quick Start

Refer to examples.

Architecture

┌────────────────────────────────────────────────────────────────────────────────—┐
│                              Snapshotter API                                    │
│                    ┌──────────────┐       ┌──────────────┐                      │
│                    │  Snapshot()  │       │  Restore()   │                      │
│                    └──────┬───────┘       └──────┬───────┘                      │
└───────────────────────────┼──────────────────────┼──────────────────────────────┘
                            │                      │
┌───────────────────────────┼──────────────────────┼──────────────────────────────┐
│                           │   Internal Layer     │                              │
│                           ▼                      ▼                              │
│  ┌─────────────────────────────────────────────────────────────────┐            │
│  │                        Metadata Manager                         │            │
│  │                         (bbolt DB)                              │            │
│  │  ┌───────────────┐  ┌──────────────┐  ┌──────────────────┐      │            │
│  │  │   Snapshots   │  │   Contents   │  │ Content Metadata │      │            │
│  │  │   Metadata    │  │   Metadata   │  │   (References)   │      │            │
│  │  └───────────────┘  └──────────────┘  └──────────────────┘      │            │
│  └─────────────────────────────────────────────────────────────────┘            │
│                                                                                 │
│  ┌─────────────────────────────────────────────────────────────────┐            │
│  │                      Storage Manager                            │            │
│  │  ┌─────────────────┐           ┌──────────────────────┐         │            │
│  │  │ Content Storage │           │  Snapshot Storage    │         │            │
│  │  │  (Sparse Files) │◄─hardlink─│  (Hardlinked Files)  │         │            │
│  │  │   [xxh3 hash]   │           │    [read-only]       │         │            │
│  │  └─────────────────┘           └──────────────────────┘         │            │
│  └─────────────────────────────────────────────────────────────────┘            │
│                                                                                 │
│  ┌──────────────────────┐                                                       │
│  │  Garbage Collector   │ (Background)                                          │
│  │  - Disk monitoring   │                                                       │
│  │  - LRU eviction      │                                                       │
│  │  - Orphan cleanup    │                                                       │
│  └──────────────────────┘                                                       │
└─────────────────────────────────────────────────────────────────────────────────┘
                            │                      │
┌───────────────────────────┼──────────────────────┼──────────────────────────────┐
│                           │  External Services   │                              │
│                           ▼                      ▼                              │
│  ┌──────────────────────────────┐    ┌─────────────────────────────┐            │
│  │      OCI Registry            │    │    Dragonfly P2P Network    │            │
│  │  ┌────────────────────────┐  │    │  ┌───────────────────────┐  │            │
│  │  │  Snapshot Manifests    │  │    │  │   Content Provider    │  │            │
│  │  │  (Metadata only)       │  │    │  │   (S3/OSS/etc.)       │  │            │
│  │  └────────────────────────┘  │    │  └───────────────────────┘  │            │
│  │  ┌────────────────────────┐  │    │  ┌───────────────────────┐  │            │
│  │  │  Config Blobs          │  │    │  │   P2P Distribution    │  │            │
│  │  │  (File metadata)       │  │    │  │   (Upload/Download)   │  │            │
│  │  └────────────────────────┘  │    │  └───────────────────────┘  │            │
│  └──────────────────────────────┘    └─────────────────────────────┘            │
└─────────────────────────────────────────────────────────────────────────────────┘

Storage Directory Structure

$RootDir/
├── metadata/               # bbolt database for metadata
├── storage/
│   ├── content/            # content-addressable storage
│   └── snapshot/           # zardlinked snapshots for read-only files

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrSnapshotAlreadyExists indicates that the snapshot already exists.
	ErrSnapshotAlreadyExists = errors.New("snapshot already exists")
)

Functions

This section is empty.

Types

type Config 

type Config struct {
	// GC is the garbage collection configuration for the snapshotter.
	GC GC

	// RootDir is the root directory for the snapshotter.
	RootDir string

	// Metadata is the metadata configuration for the snapshotter.
	Metadata Metadata

	// Content is the content configuration for the snapshotter.
	Content dragonfly.ContentProvider

	// Dragonfly is the Dragonfly configuration for the snapshotter.
	Dragonfly dragonfly.Dragonfly

	// SnapshotConcurrency is the maximum number of concurrent snapshots.
	SnapshotConcurrency *int

	// RestoreConcurrency is the maximum number of concurrent restores.
	RestoreConcurrency *int
}

Config defines the configuration for the snapshotter.

type File 

type File struct {
	// RelativePath is the relative path of the file.
	RelativePath string

	// ReadOnly indicates whether the file is read-only.
	ReadOnly bool
}

File defines the file information for snapshotting and restoring.

type GC 

type GC struct {
	// Interval is the interval at which garbage collection is performed.
	Interval *time.Duration

	// DiskHighThresholdPercent is the percentage of disk space that triggers garbage collection.
	DiskHighThresholdPercent *float64

	// DiskLowThresholdPercent is the percentage of disk space that triggers garbage collection.
	DiskLowThresholdPercent *float64

	// BatchSize is the number of snapshots to process in each batch.
	BatchSize *int

	// MinRetentionPeriod is the minimum retention period for snapshots.
	MinRetentionPeriod *time.Duration
}

GC defines the garbage collection configuration for the snapshotter.

type Metadata 

type Metadata struct {
	// Registry is the OCI registry configuration for the snapshotter.
	Registry oci.Registry
}

Metadata defines the metadata configuration for the snapshotter.

type Option 

type Option func(*Config)

Option is a function that configures the snapshotter.

func WithContent 

func WithContent(content dragonfly.ContentProvider) Option

WithContent configures the snapshotter with content.

func WithMetadata 

func WithMetadata(metadata Metadata) Option

WithMetadata configures the snapshotter with metadata.

type RestoreRequest 

type RestoreRequest struct {
	// Name is the name of the snapshot.
	Name string

	// Version is the version of the snapshot.
	Version string

	// OutputDir is the directory where the files will be restored.
	OutputDir string
}

RestoreRequest defines the request parameters for restoring files from a snapshot.

type SnapshotRequest 

type SnapshotRequest struct {
	// Name is the name of the snapshot.
	Name string

	// Version is the version of the snapshot.
	Version string

	// BaseDir is the base directory of the snapshot.
	BaseDir string

	// Files is the list of files to be included in the snapshot.
	Files []File
}

SnapshotRequest defines the request parameters for creating a snapshot.

type Snapshotter 

type Snapshotter interface {
	// Snapshot creates a snapshot of the specified files.
	Snapshot(ctx context.Context, req *SnapshotRequest, opts ...Option) error

	// Restore restores the specified files from the snapshot.
	Restore(ctx context.Context, req *RestoreRequest, opts ...Option) error
}

Snapshotter is an interface that defines the methods for snapshotting and restoring files.

func New 

func New(config Config, opts ...Option) (Snapshotter, error)

New creates a new snapshotter instance.

Source Files

View all Source files

Directories

Path Synopsis
internal
gc
* Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
metadata
* Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
storage
mocks
metadata
storage
pkg
dragonfly
* Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
fs
* Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
oci
* Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2025 The Dragonfly Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
sparsefile

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.