如何在BigQuery中缩放转轴？

执行STEP 3两次（调整 HAVING aid> NNNN和援助< NNNN ），我们得到三个表 mydataset.pivot_2001_4000 ， mydataset.pivot_4001_6000 正如你所看到的 - mydataset.pivot_1_2000预期的模式，但从1到2001年的援助功能; mydataset.pivot_2001_4000仅具有2001年至4000年的援助功能;等等

第4步 - 将所有分区数据透视表合并到最终数据透视表中，并将所有特性表示为一个表中的列

与上述步骤相同。首先，我们需要生成查询，然后运行它所以，最初我们将缝合mydataset.pivot_1_2000和mydataset.pivot_2001_4000。然后结果为mydataset.pivot_4001_6000

应该运行上面的输出字符串并将结果写入 mydataset.pivot_1_4000

然后我们重复如下所示的STEP 4

将结果写入 mydatas et.pivot_1_6000

结果表格具有以下模式：

注意： a 。我尝试了这种方法，但最多只能使用6000种功能，并且按照预期的那样工作 - b 。第3步和第4步中的第二个/主要查询的运行时间从20分钟到60分钟不等 c 。重要提示：步骤3和步骤4中的计费层从1到90不等。好消息是相应的表的大小相对较小（30-40MB），计费字节也是如此。对于2016年以前的项目，所有项目都被视为第1级，但2016年10月之后，这可能会成为问题。 更多信息，请参见时间 href =cloud.google/bigquery/pricing#high-compute =noreferrer>高级计算查询 d 。以上例子展示用BigQuery进行大规模数据转换的功能！尽管如此，我认为（但我可能错了）存储物化特征矩阵并不是最好的想法

Let's say, I have music video play stats table mydataset.stats for a given day (3B rows, 1M users, 6K artists). Simplified schema is: UserGUID String, ArtistGUID String

I need pivot/transpose artists from rows to columns, so schema will be: UserGUID String, Artist1 Int, Artist2 Int, … Artist8000 Int With Artist plays count by respective user

There was an approach suggested in How to transpose rows to columns with large amount of the data in BigQuery/SQL? and How to create dummy variable columns for thousands of categories in Google BigQuery? but looks like it doesn’t scale for numbers I have in my example

Can this approach be scaled for my example?

I tried below approach for up to 6000 features and it worked as expected. I believe it will work up to 10K features which is hard limit for number of columns in a table

STEP 1 - Aggregate plays by user / artist

STEP 2 – Normalize uid and aid – so they are consecutive numbers 1, 2, 3, … . We need this at least for two reasons: a) make later dynamically created sql as compact as possible and b) to have more usable/friendly columns names

Combined with first step – it will be:

Let’s write output to table - mydataset.aggs

STEP 3 – Using already suggested (in above mentioned questions) approach for N features (artists) at a time. In my particular example, by experimenting, I found that basic approach works well for number of features between 2000 and 3000. To be on safe side I decided to use 2000 features at a time

Below script is used for dynamically generating query that then run to create partitioned tables

Above query produces yet another query like below:

This should be run and written to mydataset.pivot_1_2000

Executing STEP 3 two more times (adjusting HAVING aid > NNNN and aid < NNNN) we get three more tables mydataset.pivot_2001_4000, mydataset.pivot_4001_6000 As you can see - mydataset.pivot_1_2000 has expected schema but for features with aid from 1 to 2001; mydataset.pivot_2001_4000 has only features with aid from 2001 to 4000; and so on

STEP 4 – Merging all partitioned pivot table to final pivot table with all features represented as columns in one table

Same as in above steps. First we need generate query and then run it So, initially we will "stitch" mydataset.pivot_1_2000 and mydataset.pivot_2001_4000. Then result with mydataset.pivot_4001_6000

Output string from above should be run and result written to mydataset.pivot_1_4000

Then we repeat STEP 4 like below

Result to be written to mydataset.pivot_1_6000

The resulted table has following schema:

NOTE: a. I tried this approach only up to 6000 features and it worked as expected b. Run time for second/main queries in step 3 and 4 varied from 20 to 60 min c. IMPORTANT: billing tier in steps 3 and 4 varied from 1 to 90. The good news is that respective table’s size is relatively small (30-40MB) so does billing bytes. For "before 2016" projects everything is billed as tier 1 but after October 2016 this can be an issue. For more information, see Timing in High-Compute queries d. Above example shows power of large-scale data transformation with BigQuery! Still I think (but I can be wrong) that storing materialized feature matrix is not the best idea