warcindexer

package module

v0.0.0-...-075ebfc Latest Latest Go to latest Published: Sep 11, 2022 License: MIT Imports: 26 Imported by: 2

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/crossedbot/go-warc-indexer

Links

Open Source Insights

Documentation ¶

Index ¶

Constants
Variables
type Indexer
- func New(ctx context.Context, addr string) (Indexer, error)
type JsonBlock

Constants ¶

View Source

const (
	// Encryption Types
	NoneEncryption   = "none"
	AesGcmEncryption = "aes-gcm"
)

Variables ¶

View Source

var (
	// CDXJ Versioning
	VersionKey   = "WARC-CDXJ"
	VersionValue = "1.0"
)

Functions ¶

This section is empty.

Types ¶

type Indexer ¶

type Indexer interface {
	// Index indexes the WARC file at the given file path and returns a CDXJ
	Index(name string) (simplecdxj.CDXJ, error)

	// SetEncryptionKey sets the encryption key and salt used to encrypt data
	// before pushing to IPFS
	SetEncryptionKey(key, salt []byte)
}

Indexer represents an interface to a WARC file to IPFS indexer

func New ¶

func New(ctx context.Context, addr string) (Indexer, error)

New returns a new Indexer, using the given Multiaddr formatted address to interact with the IPFS node

type JsonBlock ¶

type JsonBlock struct {
	// Defined Fields
	Uri string `json:"uri"`
	Ref string `json:"ref"`
	Sha string `json:"sha"`
	Hsc int    `json:"hsc"`
	Mct string `json:"mct"`
	Rid string `json:"rid"`

	// Custom Fields
	Locator          string `json:"x_locator"`
	Title            string `json:"x_title,omitempty"`
	EncryptionKeyID  string `json:"x_encryption_key_id,omitempty"`
	EncryptionMethod string `json:"x_encryption_method,omitempty"`
	EncryptionNonce  string `json:"x_encryption_nonce,omitempty"`
}

JsonBlock represents the content block (AKA payload) of the CDXJ record

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL