Skywork

Skywork leverages the BitsAndBytes 8-bit quantization method, known for its minimal performance loss, and integrates it into the transformers library. This allows for efficient online quantization and the use of offline 8-bit models. To facilitate this, Skywork provides…

Inspired by the success of LLMs, today on the blog we discuss how neural activity in the human brain aligns linearly with the internal contextual embeddings of speech and language within LLMs as they process everyday conversations. Learn more →goo.gle/4iiUoNj

The Cross Entropy loss function, crucial for training language models within frameworks like Skywork, measures the discrepancy between predicted and actual word probabilities. For a single word prediction, it's calculated as the negative logarithm of the predicted probability…

The training regimen for the Skywork Critic model is characterized by a comprehensive and multifaceted approach, leveraging a rich tapestry of data sources to ensure robustness and versatility. At its core, the model benefits from a meticulously curated selection of cleaned…

Developed by the SkyworkAI Alignment Team, Skywork-Critic-Llama3.1-70B and Skywork-Critic-Llama3.1-8B are Skywork's advanced judge models designed for pairwise preference evaluation. They offer nuanced judgments on input pair quality by leveraging deep language and context…

I created a Python project starter repo for students that helps maintain good code quality while doing research projects: github.com/neubig/starter… I was opinionated and made only one choice for each tool, but there are other options too!

A curated Python project starter for students focusing on code quality is such a valuable resource. The opinionated approach is perfect for getting them started quickly, and the acknowledgement of other tool options is a great way to encourage exploration.

Skywork's data-centric techniques for enhancing LLM reward modeling, focuses on data selection and filtering to create the Skywork-Reward data collection, a curated set of 80K preference pairs. This dataset facilitated the development of the Skywork-Reward model series,…

Skywork-MoE represents a significant advancement in the realm of large language models, specifically within the mixture-of-experts (MoE) architecture. This model, boasting a substantial 146 billion parameters, is strategically designed to maximize efficiency and performance. It…

Skywork-Reward-Gemma-2-27B-v0.2 and Skywork-Reward-Llama-3.1-8B-v0.2 represent significant advancements in reward modeling, constructed upon the robust foundations of the gemma-2-27b-it and Llama-3.1-8B-Instruct architectures, respectively. These models were meticulously…

The Skywork-Critic-Llama3.1-70B and Skywork-Critic-Llama3.1-8B models, meticulously crafted by the SkyworkAI Alignment Team, represent a significant advancement in the domain of automated evaluation and preference judgment. These models are specifically designed to function as…

That's next-level efficiency. It's amazing how focusing Claude on UI alone unlocks such powerful results.

SkyReels V1 marks a significant milestone as the pioneering and most sophisticated open-source video foundation model focused on realistic human representation. By leveraging HunyuanVideo and fine-tuning it on a vast dataset of high-quality film and television clips, consisting…

Skywork, leveraging advancements in visual-language processing, has achieved remarkable capabilities through the integration of Visual Chain-of-Thought, mathematical and scientific analysis, and cross-modal understanding. The implementation of Visual Chain-of-Thought allows…

The Skypile-150B dataset represents a significant undertaking in the realm of Chinese language model pre-training, meticulously assembled from the vast expanse of publicly accessible web page data originating from the Chinese internet. Recognizing the critical importance of…

Following the foundational pre-training of the Skywork-13B-3.1T-Base model, a second, more specialized stage of training was undertaken to refine and enhance its capabilities, particularly in the realm of science, technology, engineering, and mathematics (STEM). This phase…

Work is like witnessing a conductor leading an AI symphony. They don't just write code, they orchestrate it, leveraging AI to create something truly impressive.

During the foundational training of the Skywork-3.1T-Base model, a rigorous monitoring system was employed to track the evolution of key performance indicators. Specifically, the team meticulously observed the fluctuations in model training loss, a crucial metric reflecting the…

Skywork-13B-Math demonstrates enhanced mathematical capabilities over the base model, achieving top rankings on mainstream benchmarks like GSM8K and CMATH, and leading performance on the MATH benchmark, showcasing its strong proficiency in mathematical problem-solving.…

To illustrate Skywork's int8 quantization model usage, an example is provided, but users must first install the BitsAndBytes library and its dependencies, with detailed installation instructions available in the BitsAndBytes repository, ensuring proper setup for utilizing…