Datasources

Fetcher reads from five database engines. The job request looks the same for all of them. What differs is how each engine names its tables, how Fetcher discovers a schema, and how values come back. This page covers those differences. Pick your engine and read its section.

​

What every engine shares

---

Set the engine with the connection type, upper-case: POSTGRESQL, MYSQL, ORACLE, SQL_SERVER, MONGODB. Request validation matches those five strings exactly, so any other casing fails with 400. These bounds are the same everywhere: The four relational engines share one connection pool profile: 25 open connections, 10 idle connections, a 5-minute connection lifetime, and a 1-minute idle timeout. MongoDB uses a pool of at most 100 and at least 10, with the same 1-minute idle timeout. Field projection also works the same way. A list of field names selects those fields. The single entry ["*"] selects all of them.

​

PostgreSQL

---

Schemas. PostgreSQL is the fullest multi-schema engine in Fetcher. Discovery reads information_schema for the schemas your job names, and falls back to public when the job names none. Table names. A table outside public comes back schema-qualified, such as accounting.invoices. A table inside public comes back bare, such as users. Write the name the same way in mappedFields. Filter keys. A filter key resolves in three forms: the exact name, the name without its schema prefix, and the name with public added. transactions and public.transactions therefore reach the same table. JSON columns. The driver returns JSON and JSONB columns as raw bytes. Fetcher tries to read each value as a JSON object, then as a JSON array, then as a JSON string. The parsed value goes into the result. A value that matches none of the three stays raw, and Fetcher logs a warning.

​

MySQL

---

No schema list. MySQL is the one relational engine with no schema dimension in Fetcher. The connected database is the namespace. Discovery reads information_schema for tables, columns, and primary keys of that database. Table names. Write table names bare. A schema prefix has nothing to resolve against. Filter keys. A filter key must match the table name exactly as you wrote it under mappedFields. JSON columns. Fetcher applies the same three-step JSON read to byte values that PostgreSQL gets: object, then array, then string.

​

Oracle

---

Owners, not schemas. Oracle calls the namespace an owner, and Fetcher treats the owner as the schema. When your job names owners, discovery reads all_tables, all_tab_columns, and all_cons_columns restricted to those owners. When it names none, discovery falls back to user_tables and user_cons_columns, which cover the connected user only. The default owner is the user the connection authenticates as. Table names. A table owned by the connected user comes back bare. Any other table comes back owner-qualified, as OWNER.TABLE. Fetcher compares owner names without case sensitivity, and upper-cases the owner before it queries the Oracle catalog. Fetcher also upper-cases table and column names in the schema it reports back, and extracted rows use those upper-case column names as keys. You can write a job in upper case or lower case, because Fetcher upper-cases the names in your request the same way. Read the results by the upper-case key. Filter keys. Oracle matches a filter key exactly. There is no prefix fallback. JSON columns. Byte values pass through the same three-step JSON read.

​

SQL Server

---

Schemas. SQL Server is multi-schema, with dbo as the default. Discovery reads information_schema.tables for the schemas your job names, and restricts to dbo when the job names none. Table names. A table outside dbo comes back schema-qualified, such as accounting.invoices. A table inside dbo comes back bare. This is the same rule PostgreSQL applies to public. Filter keys. SQL Server resolves a filter key in the same three forms PostgreSQL does, with dbo as the default prefix. JSON columns. Byte values pass through the same three-step JSON read.

​

MongoDB

---

Collections, not tables. MongoDB has no schema namespace. Write collection names bare. A filter key must match the collection name exactly. Inferred schema. Documents carry no declared schema, so Fetcher builds one in two passes over each collection:

1. An aggregation pipeline collects the full set of field names.
2. A sample of up to 50 documents infers the type of each field. A field the sample never sees keeps the type unknown.

Sampling. The field-name pass adapts to collection size. Up to 10,000 documents, it aggregates over at most 1,000 documents. Above 10,000 documents it switches to a $sample stage, at a size that targets 95% confidence with a 5% margin of error: An inferred schema is therefore a strong sample, not a guarantee. A field that exists in a handful of documents of a very large collection can be missed. If the aggregation fails on a collection, Fetcher logs a warning and falls back to sampling instead of failing discovery. Filters. MongoDB takes the same ten operators, translated into query operators. A like pattern becomes a case-insensitive regular expression. MongoDB is also the one engine that does not run the UUID check on identifier-like field names. See Filters. Connection options. An internal datasource passes driver options such as authSource and directConnection through DATASOURCE_{NAME}_OPTIONS.

​

Next steps

---