Ask HK: How are you building AI apps today?

Building AI applications today involves a structured approach that goes from problem definition to continuous monitoring, leveraging a mix of established tools and emerging frameworks. While the core process is universally applicable, specific regional nuances and regulatory environments, such as those in Hong Kong, significantly influence technology choices and deployment strategies.

1. The Foundational Process: From Concept to Maintenance

At its heart, AI app development follows a clear lifecycle:

• Problem Definition and Planning: Clearly identify the problem, its requirements, and the type of AI best suited (e.g., NLP, computer vision). This initial step dictates subsequent choices.
• Data Collection and Preparation: AI models are inherently data-driven. This involves gathering substantial, relevant data from diverse sources, followed by rigorous cleaning, handling missing values, and preprocessing to ensure data quality and suitability for training.
• Model Selection and Development: Choose an appropriate model architecture, ranging from simpler machine learning algorithms to complex deep neural networks. Utilize frameworks like TensorFlow, PyTorch, or Scikit-learn for building and training these models.
• Model Evaluation and Validation: Rigorously evaluate the trained model's performance using metrics like accuracy, precision, and recall on a validation dataset. Cross-validation is a common technique here.
• Deployment: Once satisfactory, deploy the model. This can involve cloud platforms for scalability or edge devices for real-time, low-connectivity scenarios.
• Monitoring and Maintenance: Post-deployment, continuous monitoring of performance and accuracy is critical. Models can 'drift' over time, necessitating regular updates and retraining to maintain effectiveness.

2. Key Technology Stacks and Tooling

Most AI app development does not involve reinventing core AI kernels but rather shrewdly leveraging existing powerful tools and services:

• Programming Languages: Python dominates due to its rich ecosystem of libraries and strong community support.
• Machine Learning/Deep Learning Frameworks: TensorFlow and PyTorch are the go-to choices for model development.
• Development Environments: Jupyter Notebooks are widely used for exploratory data analysis and rapid prototyping.
• Version Control: Git is essential for managing code and model versions throughout the development process.
• Inference Backbones: For optimized performance, especially in environments like Hong Kong, teams utilize tools like vLLM, TensorRT-LLM, or Ollama to reduce latency and boost transactions per second.
• Vector Stores: Solutions such as Milvus, Weaviate, and Redis with vector extensions are crucial, particularly for retrieval-heavy applications common in legal, retail, and finance sectors.
• Fine-tuning: Techniques like LoRA, QLoRA, and DeepSpeed are preferred for efficient model adaptation, minimizing computational costs.
• API Wrappers: LangChain and LlamaIndex are widely adopted for speed and to reduce the need for custom 'glue code'.
• Observability: Tools like LangSmith, Arize, or custom Prometheus/Grafana setups are vital for monitoring performance, debugging, and tracking cost-per-call metrics demanded by stakeholders.
• Deployment Platforms: Cloud services like AWS, Google Cloud, and Azure are popular, offering pre-trained models and AutoML capabilities. Specific regions may prefer particular cloud zones (e.g., AWS ap-east-1). On-premise or co-location solutions (e.g., NVIDIA A100 clusters in Equinix HK Cyberport) are also common for data residency requirements in finance and law.

3. Build vs. Buy Decisions

Organizations often prioritize buying off-the-shelf solutions, especially for vertical-specific applications (e.g., smart retail, wealth management), and only build in-house when:

• Intellectual property is core to the business strategy.
• Regulatory risks necessitate proprietary development to ensure compliance (e.g., MAS 659 in Singapore might prohibit open-source model usage, requiring in-house prompt layer development).

4. Advanced Engineering: Prompts and RAG

Professional-grade applications, particularly in regulated industries, emphasize rigorous prompt and Retrieval Augmented Generation (RAG) engineering:

• Prompt Template Management: Utilizing structured banks (e.g., Google Sheets, Airtable) to manage and version prompt templates.
• RAG over Proprietary Data: Implementing RAG over internal, closed documents (e.g., ElasticSearch on PDFs) rather than public web crawls.
• Fine-tuned Rerankers: Employing custom or region-specific pretrained models (e.g., from HKUST) to improve the relevance of retrieved context chunks.
• Adversarial Testing: Conducting thorough testing for jailbreaks and hallucinations using region-specific adversarial prompts (e.g., based on WeChat's public Chinese prompts).

Outputs often include detailed prompt sheets indicating hallucination rates per domain, treated as formal UAT documents.

5. Team Structure and Talent

While AI roles are global, skill sets are tailored locally:

• Prompt Engineers: Often linguistics graduates with AI exposure.
• RAG Engineers: Strong CS background, often from top regional universities.
• ML Ops: Typically experienced professionals, sometimes from quantitative finance backgrounds.
• Business Analysts: Crucial for connecting technical development with business needs, often holding relevant certifications.

Salaries can be high for specialized roles, leading to a strong service-oriented market where prototypes are common.

6. Funding and Traction

Funding patterns vary, with seed rounds for niche open-source solutions and Series A for enterprise SaaS. However, larger funding rounds may face delays due to market-specific uncertainties.

Real-world applications demonstrate clear ROI, such as RAG solutions reducing travel time by 23% for maintenance manuals or significantly cutting compliance report drafting time.

7. Regulatory and Ethical Considerations

Building AI apps comes with significant challenges:

• Data Privacy: Navigating regulations like PIPEDA or local data laws (CSL/DSL) often requires air-gapped clusters or in-country model hosting with encrypted weights.
• Export Controls: Restrictions (e.g., China Export Rules) can impact the use and transfer of model weights, sometimes necessitating offline caching of quantized models.
• Banking Secrecy: Financial services require dual-stack architectures (public cloud + in-house servers) to prevent LLMs from accessing personally identifiable information.
• Ethics: Ensuring compliance with ethical standards and addressing concerns around model explainability.

8. The Future Roadmap

Looking ahead, expect to see more specialized applications:

• Localized Model Fine-tuning: For instance, models fine-tuned on regional financial reports.
• Embedded AI: RAG integrated into popular local platforms (e.g., WeChat Mini-programs).
• Sustainable AI: Focus on carbon-aware inference and green tariffs.
• Agentic Automation: Developing AI agents for tasks like insurance claims triage.

Navigating these complexities requires a blend of technical prowess, strategic build-or-buy decisions, and a keen awareness of the evolving regulatory and ethical landscape.