SAE 4 級和 5 級自動駕駛汽車的驗證挑戰(zhàn), 一直是駕駛模擬專家 rFpro 公司的主要關(guān)注點(diǎn)，fFpro技術(shù)總監(jiān) Chris Hoyle 表示，網(wǎng)聯(lián)自動駕駛汽車的出現(xiàn)為汽車行業(yè)帶來了一系列新的未知數(shù)。未來，汽車制造商必將建立包含數(shù)千個模擬測試場景的數(shù)據(jù)庫，以應(yīng)對汽車驗證的挑戰(zhàn)。
Hoyle 認(rèn)為，目前必須回答的關(guān)鍵問題包括：我們?nèi)绾尾拍艽_定網(wǎng)聯(lián)自動駕駛汽車可以在任何條件下都可安全運(yùn)行？我們該如何保證測試的全面性和嚴(yán)謹(jǐn)性，但又同時滿足對測試周期和成本要求？我們是否有方法在進(jìn)入驗證階段前，加速自動駕駛車輛的開發(fā)，但又同時避免給公共道路用戶帶來危險？
Hoyle 表示自己經(jīng)常遇到相關(guān)討論。他認(rèn)為，與現(xiàn)實(shí)世界中的真實(shí)測試相比，模擬仿真測試具有覆蓋范圍大、測試周期短等優(yōu)勢，但前提是必須加以正確應(yīng)用。
新的模擬仿真平臺
rFpro 公司聲稱已經(jīng)推出了世上首款針對自動駕駛汽車模擬仿真、訓(xùn)練及開發(fā)的商用平臺，可以在“各種可以想象到的環(huán)境下”對自動駕駛汽車進(jìn)行測試。據(jù)稱，該平臺的一個關(guān)鍵特點(diǎn)是“以非常高的精度，精確復(fù)制現(xiàn)實(shí)世界的測試環(huán)境”。通過一項為期 3 年的項目，rFpro 公司已經(jīng)利用高精度掃描技術(shù)（更多信息，請點(diǎn)擊這里）將大量真實(shí)道路轉(zhuǎn)換為模擬測試場景，并最終創(chuàng)建了一個“模擬測試庫”。用戶可在庫中選擇各種不同的模擬測試場景，并控制從“天氣”到“行人”等各種變量輸入。目前，這項技術(shù)已被 2 家大型OEM、3 家自動駕駛汽車開發(fā)商及 1 條無人駕駛賽車產(chǎn)品線所采用，但出于商業(yè)機(jī)密方面的考慮，rFpro 公司并未透露更多細(xì)節(jié)。
憑借一組全天候工作的計算機(jī)，制造商每月可累計數(shù)百萬英里的模擬測試?yán)锍?。Hoyle 解釋說：“從統(tǒng)計學(xué)角度而言，人類駕駛員平均每行駛 1 億英里，就會發(fā)生一起交通致死事故（來源：NHTSA數(shù)據(jù)），但我們在自動駕駛汽車的模擬測試中很難真正累計到這種級別的測試?yán)锍虜?shù)。事實(shí)上，人類駕駛員的“數(shù)據(jù)”如此優(yōu)秀是因為：在日常駕駛中，我們的絕大多數(shù)里程都是“無事發(fā)生”的。正因如此，我們可以消除這部分“無事故”里程，并通過模擬技術(shù)讓網(wǎng)聯(lián)自動駕駛汽車每隔幾秒鐘即遇到一次“千年一遇”的事件，進(jìn)而大幅縮短模擬測試周期。未來，汽車制造商將建立包含數(shù)千個模擬測試場景的數(shù)據(jù)庫，而自動駕駛汽車必須成功通過這些測試場景，才能通過驗證。”
每次遇到測試失敗的情況，模擬測試場景庫中都會新增幾項針對這一場景的標(biāo)準(zhǔn)測試。此外，車輛不僅必須成功通過每一項測試，而且還必須保證性能的穩(wěn)定。
Hoyle 表示，這種采用回歸邏輯不斷重復(fù)運(yùn)行的測試庫，可以確保任何對自動駕駛汽車的新改進(jìn)均不會影響現(xiàn)有功能。“為了實(shí)現(xiàn)這個目的，rFpro 不僅可以在一組設(shè)備上并行運(yùn)行多個實(shí)驗，而且還可在多個 CPU 和 GPU 上對一項實(shí)驗進(jìn)行擴(kuò)展，從而應(yīng)對由于自動駕駛汽車數(shù)據(jù)來源多樣（包括多部攝像頭、激光雷達(dá)、雷達(dá)傳感器等）而引入的復(fù)雜性。”
標(biāo)準(zhǔn)化模擬
程度如此密集的測試需要一定時間才能達(dá)到預(yù)期的結(jié)果。但 Hoyle 預(yù)計，未來五年中，新測試場景的增加速度將降至在統(tǒng)計學(xué)角度低于人類駕駛員出錯率的程度。這時，現(xiàn)實(shí)世界中的真實(shí)物理驗證流程就可以啟動了。
Hoyle 認(rèn)為，如果進(jìn)展理想，汽車行業(yè)將開發(fā)一套全球性的標(biāo)準(zhǔn)化測試場景庫，任何自動駕駛車型一旦通過該標(biāo)準(zhǔn)庫的驗證，即可進(jìn)入下一階段測試——即從庫里抽樣選擇一部分場景，進(jìn)行現(xiàn)實(shí)世界中的真實(shí)測試。
但這就帶來了另一個問題：我們?nèi)绾尾拍鼙ＷC這個場景庫的全面性和嚴(yán)謹(jǐn)性？ Hoyle 表示，諷刺的是，人類非常善于測試自動駕駛汽車，“因為人類具有隨機(jī)且不可預(yù)知的特點(diǎn)，從來不會重復(fù)；人類會犯錯，而且表現(xiàn)也會隨著情緒和疲勞程度的不同而改變。”目前，單個模擬測試中可以支持的人類駕駛員數(shù)量已經(jīng)增加至 50 個。我們可以測試自動駕駛汽車在人口或道路用戶密集的城市中心地區(qū)的表現(xiàn)，而且無需承擔(dān)人員傷亡的風(fēng)險。
rFpro 公司預(yù)計，到今年晚些時候，加入單獨(dú)測試的人類駕駛員數(shù)量將增加至 250 名，負(fù)責(zé)對一輛或多輛網(wǎng)聯(lián)自動駕駛汽車的測試。
人工智能 (AI) 系統(tǒng)的高效開發(fā)，離不開在啟動重新測試前從失敗中進(jìn)行學(xué)習(xí)、總結(jié)、改進(jìn)的能力。Hoyle 強(qiáng)調(diào)說，我們還會將頻繁遇到的邊緣案例（其中一個參數(shù)超過系統(tǒng)限制）或極端案例（其中兩個或兩個以上參數(shù)超過系統(tǒng)限制）反饋至系統(tǒng)中，從而不斷豐富系統(tǒng)的知識庫。
我們可以建立訓(xùn)練數(shù)據(jù)庫，用以展示失敗測試中本應(yīng)表現(xiàn)出來的正確行為，且每個數(shù)據(jù)庫均應(yīng)包含來自所有傳感器模型的數(shù)據(jù)，包括虛擬攝像頭、激光雷達(dá)和雷達(dá)。Hoyle 表示，每一幀訓(xùn)練數(shù)據(jù)均應(yīng)與“地面實(shí)況”數(shù)據(jù)相關(guān)聯(lián)，包括語義段、實(shí)例段、光流算法、深度和標(biāo)記目標(biāo)數(shù)據(jù)，“在這種邏輯下，我們的模型可以通過監(jiān)控下的學(xué)習(xí)，針對每一種新的失敗模型，進(jìn)行相應(yīng)的改進(jìn)和適應(yīng)。”
然而Hoyle 也補(bǔ)充道，盡管如此，我們還有一個至關(guān)重要的因素必須始終銘記于心， “人類是無偏差輸入的最佳來源，因為即使在相同天氣條件下的同一條道路上，人類駕駛員也不會以完全相同的方式駕駛。此外，他們可以發(fā)現(xiàn)一些異常、激怒或意外事件，并有可能因為其他道路用戶的行為而作出不當(dāng)反應(yīng)！”
The challenges of validating autonomous vehicles designed to operate at SAE Level 4 and 5 are a major focus of driving-simulation specialist rFpro. The advent of connected autonomous vehicles (CAVs) presents the auto industry with a broad new set of unknowns that will see automakers establish "libraries" of thousands of test scenarios, said the company’s technical director, Chris Hoyle.
He presents key questions that have to be answered: “How will we know that a CAV is safe to operate under all conditions? How can we ensure that testing is sufficiently comprehensive and rigorous, yet timely and cost-effective? Even before we reach the validation stage, is there a way to accelerate the development of autonomous vehicles without the risks associated with exposing them to public road users?”
Hoyle said he is routinely exposed to this debate and believes that simulation can provide greater scope and shorter timelines than physical testing—but that it must be applied correctly.
New simulation platform
The company has launched what it claims to be the world’s first commercially available platform to train and develop autonomous vehicles in simulation and test their systems in “every scenario imaginable.” A key aspect of the platform is claimed to be the level of simulation accuracy achieved replicating the real world. A three-year program has seen the company build a library of real roads via high-precision scanning technology (see previous article). Users have control of a wide range of variables, from weather to pedestrians. The technology has been adopted by two large OEMs, three autonomous-vehicle developers and a driverless motorsport series. For commercial confidential reasons, rFpro is unable to give further details.
By using a cluster of computers 24/7, manufacturers can achieve millions of simulated miles of testing every month. Explains Hoyle: “Human drivers average one fatality in 100 million miles driven (source: www-fars.nhtsa.dot.gov/Main/index.aspx), but we cannot realistically attempt to accumulate this sort of mileage with a CAV before declaring a test to be complete. The reason a human driver scores so well is because much of the distance is uneventful; by eliminating this ‘dead’ mileage and subjecting—via simulation—the CAV to a “once-in-a-1000 year” event every few seconds, we can massively compress the timescale. Vehicle manufacturers will build libraries of thousands of simulated test scenarios, which autonomous models will have to successfully pass before they will be considered ready for validation.”
Any failed experiment typically results in several more standard tests being added to the library of simulated scenarios, each of which must be reliably passed with consistency.
Hoyle said the libraries of tests, run continuously using a regression process, would ensure that any new developments to an autonomous model do not break existing functionality. “To enable this, rFpro not only scales across a cluster of machines to run multiple experiments in parallel, but it also allows each experiment to scale across multiple CPUs and GPUs to cope with the complexity of autonomous models fed by multiple camera, LiDAR and radar sensors.”
Standardizing simulation
Testing so intensively will take time to achieve required results. But Hoyle anticipates that over the next five years, the rate at which new test scenarios are identified will fall to the point at which it can be statistically proven to be below the error rate for human drivers. At that stage, the physical validation and verification process could begin.
He expects—perhaps hopefully—that the auto industry will develop a global standardized library of test scenarios which, once the model validates them, will then move forward to the next stage: a statistical sample of those tests will be selected and expanded for physical testing in the real world.
But along comes another question: how to build a library of tests that is sufficiently comprehensive and rigorous. Ironically, Hoyle stated that humans are very good at testing autonomous vehicles: “Humans are random, unpredictable, never the same twice; we make mistakes and our performance changes with mood and fatigue level.” At present, up to 50 human drivers can be added into a single simulated experiment, piloting vehicles with the autonomous model tested in densely-populated, simulated urban environments surrounded by other road users and pedestrians, without any risk of death or injury.
By late this year, rFpro anticipates this will be scaled up to 250 human test drivers entering a single experiment, shared by one or more CAVs.
Efficient development of artificial-intelligence (AI) systems requires the ability to learn from failures and improve the functionality before re-testing, stressed Hoyle, saying edge cases (where one parameter exceeds system limits) or corner cases (where a combination of two or more parameters exceeds system limits) frequently will be encountered and fed back into the system, increasing its knowledge base.
Training datasets can be established to demonstrate correct behavior for failed experiments, each comprising all the data that is fed to sensor models for virtual cameras, LiDAR and radar. Every frame of training data is associated with “ground-truth” data, comprising semantic segmentation, instance segmentation, optical flow, depth and labelled object data: “In that way, through supervised learning, the models improve and adapt to each new failure mode,” Hoyle said.
But through all this, a vital factor must always be remembered and appreciated, he added: “Humans are the best source of unbiased inputs because they never drive in an identical manner, even when repeating a journey on the same road in the same weather conditions. Also, they can identify behavior which is unusual, irritating or unexpected and likely to promote adverse reaction from other road users!”
Author: Stuart Birch
Source: SAE Automotive Engineering Magazine
圖為在fPro 中為一個缺少道路標(biāo)記的交叉路口創(chuàng)建訓(xùn)練數(shù)據(jù)集的復(fù)雜案例。(圖片來源：rFpro)
rFpro 公司正在巴黎的一個十字路口進(jìn)行測試，可以看到這個十字路口的道路標(biāo)記比較模糊。（圖片來源：rFpro）
每一幀用于監(jiān)控學(xué)習(xí)的訓(xùn)練數(shù)據(jù)均包括語義段、實(shí)例段、光流算法、深度和標(biāo)記目標(biāo)數(shù)據(jù)，圖為一個語義分割的例子。（圖片來源：rFpro）