This Week in Databend #70

November 30, 2022 · 4 min read

PsiACE

Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: .

Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .

What's New

Check out what we've done this week to make Databend even better for you.

Features & Improvements ✨

Format

better checking of format options (#8981)
add basic schema infer for parquet (#9043)

Query

QualifiedName support 'db.table.' and 'table.' (#8965)
support bulk insert without expression (#8966)

Storage

add cache layer for fuse engine (#8830)
add system table system.memory_statistics (#8945)
add optimize statistic ddl support (#8891)

Code Refactoring 🎉

Base

remove common macros (#8936)

Format

TypeDeserializer get rid of FormatSetting (#8950)

Planner

refactor extract or predicate (#8951)

Processors

optimize join by merging build data block (#8961)

New Expression

allow sparse column id in chunk, redo #8789 with a new approach. (#9008)

Documentation 📔

i18n support with crowdin (#8987, #8997, etc.)

Bug Fixes 🔧

Base

try fix lost tracker (#8932)

Meta

fix share db bug, create DatabaseIdToName if need (#9006)

Mysql handler

fix mysql conns leak (#8894)

Processors

try fix update list memory leak (#9023)

Storage

read and write block in parallel when compact (#8921)

What's On In Databend

Stay connected with the latest news about Databend.

Infer Schema at a Glance

You usually need to create a table before loading data from a file stored on a stage or somewhere. Unfortunately, sometimes you might not know the file schema to create the table or are unable to input the schema due to its complexity.

Introducing the capability to infer schema from an existing file will make the work much easier. You will even be able to query data directly from a stage using a SELECT statement like select * from @my_stage.

INFER 's3://mybucket/data.csv' FILE_FORMAT = ( TYPE = CSV );
+-------------+---------+----------+
| COLUMN_NAME | TYPE    | NULLABLE |
|-------------+---------+----------|
| CONTINENT   | TEXT    | True     |
| COUNTRY     | VARIANT | True     |
+-------------+---------+----------+

We've added support for inferring the basic schema from parquet files in #9043, and we're now working on #7211 to implement select from @stage.

Learn More

What's Up Next

We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.

Add Tls Support for Mysql Handler

opensrv-mysql v0.3.0 that was released recently includes support for TLS. It sounds like a good idea to introduce it to Databend.

let (is_ssl, init_params) = opensrv_mysql::AsyncMysqlIntermediary::init_before_ssl(
    &mut shim,
    &mut r,
    &mut w,
    &Some(tls_config.clone()),
)
.await
.unwrap();

opensrv_mysql::secure_run_with_options(shim, w, ops, tls_config, init_params).await

Issue 8983: Feature: tls support for mysql handler

Please let us know if you're interested in contributing to this issue, or pick up a good first issue at https://link.databend.rs/i-m-feeling-lucky to get started.